{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Los Alamos National Lab Data Preparation\n",
    "*Source:* [LANL dataset](https://csr.lanl.gov/data/cyber1/)  \n",
    "\n",
    "This data set represents 58 consecutive days of de-identified event data collected from five sources within Los Alamos National Laboratory’s corporate, internal computer network. \n",
    "\n",
    "Only the auth.txt file is used in our current work, as all red team activity appearing in the data correspond exclusivley to authentication events. Future work includes utilizing additional data streams (namely; proc.txt.gz). We preform a pre-processing step on the file redteam.txt.gz so that its log lines are expanded to match the full log line which they correspond to in the auth.txt.gz file. This adds general convenience, and speeds up the process of querying to find out if a given log line is malicious.\n",
    "\n",
    "\n",
    "### Character Level\n",
    "At the character level, the dataset is parsed line by line, and the ascii value for each character in the log line is written to an output file, which can then be fed into the model. \n",
    "\n",
    "That translation in Python is,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def translate_line(string, pad_len):\n",
    "    return \"0 \" + \" \".join([str(ord(c) - 30) for c in string]) + \" 1 \" + \" \".join([\"0\"] * pad_len) + \"\\n\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where **string** is a log line to be translated, and **pad_len** is the number of 0's to append so that the length of the translated string has the same number of characters as the longest log line in the dataset (character-wise). The start of sentence token used here is **0** and the end of sentence token is **1**. \n",
    "\n",
    "The length of the max log line can be obtained using,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "94\n"
     ]
    }
   ],
   "source": [
    "max_len = 0\n",
    "with open(\"auth.txt\", \"r\") as infile:\n",
    "    for line in infile:\n",
    "        if len(line) > max_len:\n",
    "            max_len = len(line)\n",
    "print (max_len)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We choose to filter out the weekends for this dataset, since very little happens on these days. These days are\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weekend_days = [3, 4, 10, 11, 17, 18, 24, 25, 31, 32, 38, 39, 45, 46, 47, 52, 53]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also convenient to keep track of which lines are in fact red events. Note that labels are not used during unsupervised training, this step is included to simplify the evaluation process later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(\"redevents.txt\", 'r') as red:\n",
    "    redevents = set(red.readlines())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We then loop through auth.txt line by line, and symultaneously read in (raw) and write out (translated) log lines. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "with open(\"auth.txt\", 'r') as infile, open(\"char_feats.txt\", 'w') as outfile:\n",
    "    outfile.write('line_number second day user red seq_len start_sentence\\n') # header\n",
    "    infile.readline()\n",
    "    for line_num, line in enumerate(infile):\n",
    "        line_minus_time = ','.join(line.strip().split(',')[1:])\n",
    "        diff = max_len - len(line_minus_time)\n",
    "        raw_line = line.split(\",\")\n",
    "        sec = raw_line[0]\n",
    "        user = raw_line[1].strip().split('@')[0]\n",
    "        day = math.floor(int(sec)/86400)\n",
    "        red = 0\n",
    "        red += int(line in redevents) # 1 if line is red event\n",
    "        if user.startswith('U') and day not in weekend_days:\n",
    "            index_rep = translate_line(line_minus_time, 120) # diff\n",
    "            outfile.write(\"%s %s %s %s %s %s %s\" % (line_num, \n",
    "                                                    sec, \n",
    "                                                    day, \n",
    "                                                    user.replace(\"U\", \"\"), \n",
    "                                                    red, \n",
    "                                                    len(line_minus_time) + 1,\n",
    "                                                    index_rep))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use char_feats.txt as input data for the character level models.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Level\n",
    "For word level, our approach is to build a vocab based on each unique feature token that appears in the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
